Stereoscopic video recording method, stereoscopic video recording medium, stereoscopic video reproducing method, stereoscopic video recording apparatus, and stereoscopic video reproducing apparatus

ABSTRACT

The present invention is for more optimally performing recording and reproduction of stereoscopic video. In the present invention, parallax information is utilized to store, in a recording medium, stereoscopic video including left-eye images and right-eye images. Particularly, for video content that contains stereoscopic images, information is obtained regarding the amount of variation in parallactic angle having a given or larger value, a variation time that the variation in parallactic angle takes, and the number of times that the variation in parallactic angle occurs. An evaluation value is calculated that corresponds to the degree of eye fatigue on the basis of the amount of variation, the variation time, and the number of times of the variation. According to the present invention, the video content is encoded in such a manner that the evaluation value is within a given range, and then recorded in the recording medium.

CROSS-REFERENCE OF RELATED APPLICATIONS

This application is a Continuation of U.S. patent application Ser. No.14/850,370, filed on Sep. 10, 2015, which is a Continuation of U.S.patent application Ser. No. 14/192,120, filed on Feb. 27, 2014, issuedas U.S. Pat. No. 9,167,227, on Oct. 20, 2015, which is a Continuation ofU.S. patent application Ser. No. 12/153,244, filed on May 15, 2008,which claims priority under 35 U.S.C. §1.119(a) to Japanese PatentApplication No. 2007-309084, filed on Nov. 29, 2007, all of which arehereby expressly incorporated by reference into the present application.

BACKGROUND OF THE INVENTION

Field of the Invention

The present invention relates to a stereoscopic video recording method,a stereoscopic video recording medium, a stereoscopic video reproducingmethod, a stereoscopic video recording apparatus, and a stereoscopicvideo reproducing apparatus.

Description of the Background Art

With the progress of encoding techniques for compressing digital videosignals, it has become possible to implement optical disk apparatuseswith excellent searchability and operability by recording compressedvideo signals to optical disks. Such optical disk apparatuses are freefrom degradation caused by dubbing, because they record digital videosignals instead of analog video signals, and such optical diskapparatuses offer improved reliability because they employ non-contactoptical recording and reproduction.

Encoding methods for compressing such digital video signals include theMPEG (Moving Picture coding Experts Group) standards, for example. TheMPEG compression offers improved compression efficiency as compared with“intra” compression methods such as motion JPEG, but it still hasrestrictions due to the use of motion-compensated prediction in thetemporal direction; for example, searches can be made only in units ofgroups of multiple pictures (GOPs), and access on the disk requiresaccessing an intra-compressed I picture first. Accordingly, JapanesePatent Application Laid-Open No. 2005-260988 (Patent Document 1)suggests a devised data format on an optical disk.

However, the data format of Patent Document 1 has been made only forfiling of two-dimensional, flat images. For filing of stereoscopicimages, the method of Japanese Patent Application Laid-Open No.2007-166651 has to be adopted, in which a right-eye image and a left-eyeimage are separately displayed in the first and second fields of a TVsignal, and the two individual field images are viewed by the respectiveeyes through the use of means such as polarizing eyeglasses.

Viewing stereoscopic (hereinafter referred to also as 3D) video for manyhours causes discomforts such as eye fatigue and 3D motion sickness,because the viewer moves the eyes in the focal direction with increasedfrequency. Three-dimensional viewing will be problematic especially whenthe point of attention busily moves in the depth direction. Also, inparticular, the display of subtitles and the like requires continuouslyswitching the eye focus, and may cause more intense discomforts like eyefatigue and 3D motion sickness than the viewing of ordinary 3D video.

Also, as to the recording of stereoscopic video information, when filinga stereoscopic video that employs parallax between right-eye images andleft-eye images on an optical disk, it was not possible to handle avideo stream including a mixture of flat (hereinafter referred to alsoas 2D) images and stereoscopic (3D) images. Particularly, for example,when the display apparatus is switched while the reproducing apparatusis reproducing stereoscopic video, or when a display apparatus is newlyconnected, information required for 3D display cannot be sent to thedisplay apparatus side in a linked manner, and so the settings in thedisplay apparatus cannot be changed instantaneously.

Also, as to the recording of stereoscopic video information, when filinga stereoscopic video employing parallax between right-eye and left-eyeimages on an optical disk, there is no video control information thatallows recording of broadcasted stereoscopic video and that enablessettings in the display apparatus and the reproducing apparatus.

Also, distributed content, such as movies, are usually released atdifferent times, for preferentially releasing businesses such as movietheaters and for the distribution as media, and therefore “code” isprovided to limit the region where the movie can be viewed, according tothe conditions of distribution of the movie. Also, the distribution ofstereoscopic video content is affected by the extent of proliferation ofcompatible display apparatuses, and it is necessary to further setseparate regional restrictions for 3D video and for 2D video. However,conventional systems do not meet such requirements.

Also, it was not possible to superimpose information about the content,copyright information, 3D display scheme information, OSD information,multi-angle information, etc. on a stereoscopic video stream, andtherefore it was not possible to instantaneously change the settings onthe display apparatus side, not on the reproducing apparatus side.

Also, unlike ordinary 2D video information, 3D video informationutilizing parallax includes information for the left-eye and informationfor the right eye, and the amount of information is doubled and filingefficiency is deteriorated.

Also, while increased eye fatigue occurs during the reproduction of 3Dvideo with an increased degree of three-dimensionality, it was notpossible to give a warning to the user in advance, according to thedegree of three-dimensionality, or to restrict the viewing according tothe age of the user. Also, while increased eye fatigue occurs during thereproduction of 3D video with an increased degree ofthree-dimensionality, it was not possible to alleviate thethree-dimensionality of the content on the reproducing apparatus side.

SUMMARY OF THE INVENTION

An object of the present invention is to provide a stereoscopic videorecording method, a stereoscopic video recording medium, a stereoscopicvideo reproducing method, a stereoscopic video recording apparatus, anda stereoscopic video reproducing apparatus that are capable of moreoptimally performing recording and reproduction of stereoscopic video.

A stereoscopic video recording method according to the preferredembodiment of the present invention records, in a recording medium,stereoscopic video including left-eye and right-eye images utilizingparallax information. From video content that contains stereoscopicvideo, the stereoscopic video recording method of the invention graspsthe amount of variation in parallactic angle having a given or largervalue, a variation time that the variation in parallactic angle takes,and the number of times that the variation in parallactic angle occurs.The method then calculates an evaluation value that corresponds to thedegree of eye fatigue on the basis of the amount of variation, thevariation time, and the number of times of the variation, encodes thevideo content in such a manner that the evaluation value is within agiven range, and records the encoded video content in the recordingmedium.

The stereoscopic video recording method of the preferred embodiment ofthe present invention calculates an evaluation value that corresponds tothe degree of eye fatigue on the basis of the amount of variation, thevariation time, and the number of times of the variation, and encodesthe video content in such a manner that the evaluation value is within agiven range. It is thus possible to produce video content that onlyinvolves eye fatigue limited within a certain range and to provide thevideo content to users.

A stereoscopic video reproducing method according to the preferredembodiment of the present invention reproduces stereoscopic videoincluding left-eye and right-eye images utilizing parallax information.According to the stereoscopic video reproducing method of the invention,a recording medium records video content that includes stereoscopicvideo and that is encoded in such a manner that a maximum parallacticangle value grasped from the video content is not more than a givenvalue, and the stereoscopic video reproducing method reads the maximumparallactic angle value from the recording medium, and displays themaximum parallactic angle value when a user views the video content sothat the user can recognize the maximum degree of three-dimensionalityof the stereoscopic video.

The stereoscopic video reproducing method of the preferred embodiment ofthe present invention displays the maximum parallactic angle value sothat the user can recognize the maximum degree of three-dimensionalityof the stereoscopic video when viewing the video content, and it is thuspossible to objectively grasp the influence of the stereoscopic video onthe user.

A stereoscopic video recording apparatus according to the preferredembodiment of the present invention includes one video generating blockprovided for one of the left and right eyes and another video generatingblock provided for the other eye, or includes one video generating blockthat is operated at a doubled rate. Each of the one and another videogenerating blocks, or the one video generating block, includes: an ADconverter that digitizes a video signal of video content that containsstereoscopic video; a motion detecting circuit that detects motionvectors necessary to video-compress the digitized video signal in atemporal direction; a DCT transform circuit that applies DCT transformnecessary for intra-compression to the digitized video signal; anadaptive quantization circuit that applies quantization necessary forintra-compression to the DCT-transformed video signal; a variable-lengthcoding circuit that applies variable-length coding necessary forintra-compression to the quantized video signal; and an inversequantization circuit and an inverse DCT transform circuit that decodethe quantized video signal as a local decoder. The stereoscopic videorecording apparatus of the invention records the video content and itssupplementary information in a recording medium, wherein the videocontent is composed of digital video information including an I picturethat is data-compressed within a frame, a P picture that isdata-compressed with motion compensation from the I picture in apreceding direction in time, and a B picture that is data-compressedwith motion compensation from the I or P picture in apreceding/following direction in time, and in the video content, flatand stereoscopic images in units of the digital video information or anintegral multiple of the digital video information exist in a mixedmanner, and form a temporally continuous video unit.

The stereoscopic video recording apparatus of the preferred embodimentof the present invention includes video generating blocks for left-eyeimages and for right-eye images, or one video generating block that isdriven at a doubled rate, and the apparatus is capable of obtainingappropriate stereoscopic images.

A stereoscopic video reproducing apparatus according to the preferredembodiment of the present invention reproduces a recording medium thatrecords video content and supplementary information, wherein the videocontent is composed of digital video information including an I picturethat is data-compressed within a frame, a P picture that isdata-compressed with motion compensation from the I picture in apreceding direction in time, and a B picture that is data-compressedwith motion compensation from the i or P picture in apreceding/following direction in time, and in the video content, flatimages and stereoscopic images utilizing parallax information in unitsof the digital video information or an integral multiple of the digitalvideo information exist in a mixed manner, and form a temporallycontinuous video unit. The stereoscopic video reproducing apparatus ofthe invention includes: a system decoder that separates a video streamcontaining video audio data from a signal read from the recordingmedium; a compressed-video decoder that decompresses compressed video ofthe separated video stream; an audio decoder that decompressescompressed audio of the separated video stream; an OSD decoder thatextracts OSD information that contains subtitle display from thesupplementary information; a depth generating circuit that generates adepth-direction location of the OSD information from the supplementaryinformation; a stereoscopic video processing circuit that generates thestereoscopic images from the video information decompressed by thecompressed-video decoder, and a blending circuit that superimposes theOSD information on the stereoscopic images.

The stereoscopic video reproducing apparatus of the preferred embodimentof the present invention considers the location of OSD information inthe depth direction when superimposing the OSD information onstereoscopic images, and it is thus possible to further alleviate theeye fatigue caused by the viewing of the OSD information andstereoscopic images.

These and other objects, features, aspects and advantages of the presentinvention will become more apparent from the following detaileddescription of the present invention when taken in conjunction with theaccompanying drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a conceptual diagram of a stereoscopic video display systemusing a shutter according to a first preferred embodiment of the presentinvention;

FIG. 2 is a conceptual diagram of a stereoscopic video display systemusing polarizing filters according to the first preferred embodiment ofthe present invention;

FIG. 3 is a conceptual diagram of a stereoscopic video display systemusing a rotary polarizing filter according to the first preferredembodiment of the present invention;

FIG. 4 is a conceptual diagram of a stereoscopic video display systemusing multiple information according to the first preferred embodimentof the present invention;

FIG. 5 is a schematic diagram illustrating a stereoscopic video imageaccording to the first preferred embodiment of the present invention;

FIG. 6 is a schematic diagram illustrating a stereoscopic video imageincluding display of subtitles according to the first preferredembodiment of the present invention;

FIGS. 7A and 7B are diagrams used to illustrate the degree of eyefatigue according to the first preferred embodiment of the presentinvention;

FIG. 8 is a diagram used to illustrate the degree of eye fatigue anddepth-direction acceleration according to the first preferred embodimentof the present invention;

FIG. 9 is a diagram illustrating a range of display of subtitlesaccording to the first preferred embodiment of the present invention;

FIG. 10 is a diagram illustrating information recording regions on arecording medium according to a second preferred embodiment of thepresent invention;

FIG. 11 is a diagram illustrating the structure of a video stream of avideo title according to the second preferred embodiment of the presentinvention;

FIG. 12 is a diagram illustrating the structure of the video stream ofthe video title according to the second preferred embodiment of thepresent invention;

FIG. 13 is a diagram illustrating the structure of the video stream ofthe video title according to the second preferred embodiment of thepresent invention;

FIG. 14 is a diagram illustrating content information in thesupplementary information according to the second preferred embodimentof the present invention;

FIG. 15 is a diagram illustrating time code information in thesupplementary information according to the second preferred embodimentof the present invention;

FIG. 16 is a diagram illustrating location information in thesupplementary information according to the second preferred embodimentof the present invention;

FIG. 17 is a diagram illustrating video information in the supplementaryinformation according to the second preferred embodiment of the presentinvention;

FIG. 18 is a conceptual diagram illustrating a TV display using thesupplementary information according to the second preferred embodimentof the present invention;

FIGS. 19A and 19B are schematic diagrams used to illustrate multi-angleinformation according to the second preferred embodiment of the presentinvention;

FIG. 20 is a diagram illustrating OSD information according to thesecond preferred embodiment of the present invention;

FIG. 21 is a diagram illustrating GOP table information in the videocontrol information according to the second preferred embodiment of thepresent invention;

FIG. 22 is a diagram illustrating the GOP table information and OSDinformation in the video control information according to the secondpreferred embodiment of the present invention;

FIG. 23 is a diagram illustrating sequence information, OSD information,and video attribute information in the video control informationaccording to the second preferred embodiment of the present invention;

FIG. 24 is a block diagram of a stereoscopic video recording apparatusaccording to a third preferred embodiment of the present invention;

FIG. 25 is a diagram illustrating the structure of a video streamaccording to the third preferred embodiment of the present invention;

FIG. 26 is a diagram illustrating the structure of a video streamaccording to the third preferred embodiment of the present invention;

FIG. 27 is a block diagram of a stereoscopic video recording apparatusaccording to the third preferred embodiment of the present invention;

FIGS. 28A and 28B are conceptual diagrams of parallactic images used toillustrate the principles of compression with differential informationaccording to the third preferred embodiment of the present invention;

FIG. 29 is a block diagram of a stereoscopic video recording apparatusaccording to the third preferred embodiment of the present invention;

FIG. 30 is a schematic diagram illustrating a compression by imageconversion with parallax information according to the third preferredembodiment of the present invention;

FIG. 31 is a diagram illustrating the structure of a video streamaccording to the third preferred embodiment of the present invention;

FIG. 32 is a diagram illustrating the structure of a video streamaccording to the third preferred embodiment of the present invention;

FIG. 33 is a block diagram of a stereoscopic video recording apparatusaccording to the third preferred embodiment of the present invention;

FIG. 34 is a diagram illustrating the structure of a video streamaccording to the third preferred embodiment of the present invention;

FIG. 35 is a diagram illustrating the structure of a video streamaccording to the third preferred embodiment of the present invention;

FIG. 36 is a block diagram of a stereoscopic video reproducing apparatusaccording to a fourth preferred embodiment of the present invention;

FIG. 37 is a block diagram of a stereoscopic video reproducing apparatusaccording to the fourth preferred embodiment of the present invention;

FIGS. 38A and 38B are diagrams illustrating a stereoscopic videoreproducing apparatus according to the fourth preferred embodiment ofthe present invention;

FIGS. 39A to 39D are diagrams illustrating a stereoscopic videoreproducing apparatus according to the fourth preferred embodiment; and

FIG. 40 is a diagram illustrating a stereoscopic video reproducingapparatus according to the fourth preferred embodiment.

DESCRIPTION OF THE PREFERRED EMBODIMENTS First Preferred Embodiment

A first preferred embodiment of the present invention will now bedescribed referring to the drawings. FIG. 1 is a block diagramillustrating the overall configuration of a stereoscopic video systemaccording to this preferred embodiment. The stereoscopic video systemshown in FIG. 1 includes a stereoscopic video recording apparatus 6(which will hereinafter be simply referred to also as a recordingapparatus 6) for reproducing a medium that records stereoscopic videoand outputting right-eye images 1 and left-eye images 2, a displayapparatus 3 such as a TV or a projector, a shutter 4 composed of e.g.liquid crystal and capable of switching two transmissive polarized lightrays, and eyeglasses 7A having liquid-crystal shutters on the left andright, or having different polarizing plates on the left and right, soas to view a frame sequence of video images 5A and 5B through theshutter 4. FIG. 2 illustrates another example of the configuration of astereoscopic video system of the first preferred embodiment. Unlike thatshown in FIG. 1, the stereoscopic video system of FIG. 2 includes twodisplay apparatuses A and B, polarizing plates 9 and 10 for respectivelypassing light rays having particular polarization components ofdifferent directions, an optical system 11 for projection onto a displaypanel 12, and eyeglasses 7B having different polarizing plates on theleft and right for viewing a frame sequence of video images 5C and 5Dthrough the polarizing plates 9 and 10.

FIG. 3 illustrates still another example of the configuration of astereoscopic video system of the first preferred embodiment. Unlike thatshown in FIG. 1, the stereoscopic video system of FIG. 3 includes asynchronous rotating member 13 having a circular rotary disk in whichsemi-circular polarizing plates are bonded to pass light of particularpolarization components of different directions, and a light source 14for projecting light to the synchronous rotating member 13. FIG. 4illustrates still another example of the configuration of a stereoscopicvideo system of the first preferred embodiment. Unlike that shown inFIG. 1, the stereoscopic video system of FIG. 4 includes display devices15 to 19 for projecting a plurality of video images based on videosignals 22, and a rotary mirror 21 for reproducing the projectedstereoscopic video images.

FIG. 5 is a schematic diagram illustrating the human perception ofstereoscopic video. FIG. 5 shows sequential depth positions 112 to 117from an infinite distance 111, where the depth position 117 is the mostprotruding position (closest to the eyes). FIG. 5 also shows a person118 displayed at the depth position 115, a person 119 displayed at thedepth 114, a river 120 flowing from a distance, and a mountain 121viewed at a distance. FIG. 6 shows the display of FIG. 5 with subtitlesadded thereto, which shows subtitles 122A to 122C displayed atrespective depth positions and subtitle depth ranges 123A to 123C.

FIGS. 7A and 7B are diagrams illustrating the degree of eye fatigue withrespect to “acceleration of parallactic angle variation•the time thatthe variation takes•the number of times that the variation occurs”. InFIG. 7A, the vertical axis shows the degree of eye fatigue 124, and thehorizontal axis shows the product 125 of the angular variationacceleration, the variation time, and the number of times, and FIG. 7Ashows the degree of eye fatigue 126 about a point of attention andsubtitles, a maximum-fatigue point 127 about the point of attention andsubtitles, the degree of eye fatigue 128 about the background image, alimit 129 over which the background image causes motion sickness, a saferegion 130 for human eyes, a dangerous region 131 for human eyes, and aregion 132 where human eyes suffer 3D motion sickness. FIG. 7B is aschematic diagram illustrating the parallactic movement of the eyes,which shows a far object 126A as a point of attention, a near object126B as a point of attention, a viewing angle 126D of the near object, aviewing angle 126E of the far object, human eyes 126F, a depth 126G atwhich the object 126B is displayed, and a depth 126H at which the object126A is displayed.

FIG. 8 is a graph illustrating a relation between “depth-directionacceleration of a point of attention” and “moving time×number of times”.In the graph of FIG. 8, the vertical axis shows the point-of-attentiondepth-direction acceleration 133 and the horizontal axis shows theproduct of moving time and number of times (moving time×number of times)134, and FIG. 8 shows a boundary 135 between a safe region 130 and adangerous region 131, and a boundary 136 between the dangerous region131 and a 3D motion sickness occurring region 132. FIG. 9 illustrates arelation between the depth position and the amount of depth-positionvariation of subtitles. In FIG. 9, the vertical axis shows the depthposition 137 and the horizontal axis shows the amount of depth variation140, and FIG. 9 shows an infinite distance position 138, an eye position(most forward) 139, a limit 141 of the amount of depth variation, adepth limitation 142 on the near side (a limitation of protrusion), anda depth limitation 143 on the far side.

Now, as shown in FIGS. 1 to 3, a stereoscopic video system using a TV orprojector generally utilizes parallax information about human eyes, byprojecting visual information about left and right images to provide athree-dimensional view through eyeglasses. In general, the filming ofimages uses two cameras to take pictures for the left eye and the righteye, and hence produces two streams. In such a system, the visualinformation for the left and the right stored in the recording apparatus6 is inputted to a display apparatus such as a TV or a projector. Inthis process, when the information interface between the recordingapparatus 6 and the display apparatus is analog, separate transmissionsof information are required for the left and the right. However, in thecase of a digital interface such as HDMI, the information for the leftand the information for the right can be alternately transmitted in aserial manner. Also, the information may be sent in a compressed formfrom the recording apparatus 6 and decompressed on the TV side. Also, inthe case of TV display, left image information and right imageinformation are switched field by field, in which case, during theprocess of dividing the reproduced right and left images field by field,problems like flickering can be prevented and smooth stereoscopic videoreproduction can be achieved by using a latest TV employing “scandoubling”.

As shown in FIG. 1, when the shutter 4 is composed of liquid crystalsuch that two transmissive polarized light rays can be switched, it ispossible to vary the polarizing angle of light for each field bycontrolling the shutter 4, for example with the transmitted field image5A being vertically polarized, and the image 5B being horizontallypolarized. In this case, the eyeglasses 7A are composed of differentpolarizing plates (vertical polarization and horizontal polarization)that are bonded on the right and left, and there is no need for thecable 3B from the display apparatus 3 to the eyeglasses 7A, forsupplying a signal corresponding to the timing by which the displayapparatus 3 controls the shutter 4 through the cable 3A. On the otherhand, when the shutter 4 is not used, the eyeglasses 7A have to beequipped with liquid-crystal shutters, and the cable 3B is needed as afield-synchronization signal cable. When liquid-crystal shutters areprovided on the eyeglasses 7A side, no polarized light is used, and sothe stereoscopic display is not significantly influenced even when theangle of the eyeglasses is changed, e.g. when the viewer bends his neck.

The system shown in FIG. 2 includes two display devices having PLD(Paper-like Display) elements or a transmitting liquid-crystal type, anddifferent images are displayed separately on the left and right. In thiscase, the polarizing plates 9 and 10 having different polarizingdirections are provided in front of the display devices (A and B) 7 and8. Thus, the light rays emitted from the respective display lightemitting portions have different polarizations, and the rays of lightare projected to the display panel 12 through the optical system 11 sothat the right eye views a vertically-polarized image 5C and the lefteye views a horizontally-polarized image 5D, for example. In this case,the polarizing eyeglasses 7B are used to cause the video informationhaving parallax to enter the respective eyes.

In the system of FIG. 3, the light-source portion, for emitting lightonto optical elements like PLD, includes a polarized light switchingmechanism having the synchronous rotating member 13 that rotates insynchronization with the timing of TV field display, so that lighthaving polarization adjusted to the timing of field display enters thePLD element etc. In this case, an image of different polarized light foreach field is projected to the image display panel. It is thus possibleto view images with parallax through the polarizing eyeglasses 7constructed as shown in FIG. 2. There is another method for reproducingstereoscopic images in which, as shown in FIG. 4, a plurality of displaydevices 15 to 19 are used to project images that are filmed from aplurality of angles. In this case, it is necessary to store andreproduce a plurality of stereoscopic video streams, instead of twostreams.

In another scheme for directing left and right images to thecorresponding eyes, the synchronous rotating member 13, which rotates insynchronization with the timing of TV field display, is composed ofoptical filters that pass only particular wavelengths of RGB, one halfof the disk being allocated for the left eye and the other half for theright eye, and the respective RGB wavelengths are shifted to vary thelight wavelengths for the right eye and the left eye. Also, theeyeglasses 7 are composed of optical filters that respectively pass onlythe right-eye and left-eye wavelengths. In this case, it is possible tocorrect the shifts of RGB on the left and right by color control on theTV display side, so as to obtain satisfactory color reproduction. Thisscheme, in which RGB wavelengths are shifted on the left and right, isfree from attenuation of the light from the eyeglasses 7 even when theeyeglasses are tilted.

In the system using the rotary mirror 21 as shown in FIG. 4, the displaydevices 15 to 19 project stereoscopic images from a plurality of viewpoints onto the rotary mirror 21, and the actual stereoscopic imageslook like real objects even when viewed from varied points of view (inan extreme case, a hidden portion, e.g. on the back, becomes viewable).

Next, an actual stereoscopic view will be described. Reproduced videoimages utilizing parallax are perceived as shown in FIG. 5 by humaneyes. In this case, when the depth from the infinite distance 111 to theeye position 117 is represented in an exploded manner, the individualobjects are displayed on the depth planes from the depth positions 112to 115. For example, the person as a point of attention is seen as alarge man 118 when positioned near, and it is seen as a small man 119when positioned distant. The river 121, as background information, isseen as being larger as it approaches nearer, and it is seen as beingsmaller as it goes away, while the large mountain 121 is seen as beinglarge though it forms the background. For example, when subtitles aredisplayed on the stereoscopic video as shown in FIG. 5, they arerepresented as shown in FIG. 6. The subtitle 122A positioned neargradually moves away as shown by the subtitles 122B and 122C. When theperson's depth positions 119A to 119C as a point of attention varyacross scenes, eye fatigue can be alleviated by reducing the movement ofeye focus, by varying the subtitles in coordination with the variationof the depth position. Thus, it is desirable to display the subtitle122A in the scene of the person 119A, to display the subtitle 122B inthe scene of the person 119B, and the subtitle 122C in the scene of theperson 119C. This is based on the fact that stereoscopic video, eventhough utilizing parallax, requires eye movements in correspondence withthe parallax, while conventional 2D video involves no movements of eyemuscles in the focus direction because the depth position originallydoes not vary and the distance between the user and the TV correspondsto the focus of human eyes.

As shown in FIG. 7A, as a point of attention, e.g. a person appearing inthe video, moves in the focal direction, eye fatigue occurs inproportion to the acceleration of parallactic angle variation, the timethe variation takes, and the number of times that the variation occurs.Especially, a point of attention causes intensive eye fatigue becausethe eyes have to follow it, and it is thought that the fatigue reaches apeak before the speed of parallactic angle variation, the time ofvariation, and the number of times become too large. In particular, asshown in FIG. 7B, a near object involves a larger parallactic angle asshown by the viewing angle 126D, while a distant object involves asmaller parallactic angle as shown by the viewing angle 126E.Parallactic angle varies as the distance varies, and then both eyes 126Fhave to focus to the target at an appropriate angle, in order to followthe variation of parallactic angle caused by the variation of distance.Conventional TVs for displaying flat video do not provide images thatvary in the distance direction, and therefore the parallactic anglerequired for eye perception in the depth direction always remainsconstant. However, stereoscopic video requires eye movements not only ina flat plane but also in the depth direction (with parallactic angle),resulting in an increased burden on the eyes. However, it is estimatedthat, when the video has movements faster than eye response, the degreeof eye fatigue varies as shown by the curve 126, because the eyes cannotfollow the movements and the fatigue decreases. Also, as to thebackground information, though the eyes originally do not follow it, itis estimated that the fatigue increases as the speed of parallacticangle variation, the time of variation, and the number of times increaseas the distance varies. FIG. 8 illustrates this with a relation betweenthe depth-direction acceleration and the product of moving time and thenumber of times. Even at lower depth-direction accelerations, adangerous region exists and motion sickness occurs as the number oftimes and distance increase, but it is estimated that the fatigue doesnot occur below a certain level even when the product of moving time andnumber of times increases.

As to the degree of eye fatigue, larger-sized screens require increasedeye movements in the plane direction and therefore cause increased eyefatigue. Two evaluation functions are possible accordingly, one withscreen-size consideration and the other with no screen-sizeconsideration. First, with an evaluation function 1, a value “a” atwhich the eyes' following movements can be ignored<parallactic anglevariation speed of a point of attention<a range “b” of the eyes'following movements, and the evaluation value (the degree of eyefatigue) is proportional to the value of “parallactic angle variationspeed×variation time×the number of times”. With an evaluation function2, a value “a” at which the eyes' following movements can beignored<parallactic angle variation speed of a point of attention<arange “b” of the eyes' following movements, and the evaluation value(the degree of eye fatigue) is proportional to the value of “parallacticangle variation speed×variation time×the number of times×screen size”.The evaluation function 2 is used when the TV screen size is detectable,and the evaluation function 1 is used when it is not detectable. In thesecond and following preferred embodiments, the evaluation value (thedegree of eye fatigue) is described as the degree of depth variation.

In the production of one piece of stereoscopic video, “the amount ofvariation of parallactic angle”, “the time that the variation takes”,and “the number of times that the variation occurs” are provided asevaluation factors for the stereoscopic video in that one piece of videocontent, and it is then possible to make stereoscopic video content byre-encoding it such that the value does not enter the dangerous region131 of FIG. 7. Also, the degree of depth variation, as an evaluationfunction, is described in the stereoscopic video content, and it is thenpossible to present the degree of eye fatigue before the user views themovie, so as to prompt the user to select between 2D reproduction and 3Dreproduction. In this case, examples of the re-encoding include: afilming scheme in which parallactic images are filmed with cameraslocated at a reduced parallactic interval (the distance between the twocameras is reduced); an image-processing scheme in which parallax isreduced by pixel conversion processing using parallax information, aswill be described later, and a scheme in which the amount of protrusion(the degree of three-dimensionality) is limited in the production ofcontent by CG for animation etc.

As to the display of subtitles that the user necessarily has to read tounderstand the story of the movie or the like, it is necessary to placea limitation on the amount of depth-direction variation as shown in FIG.9. This is because of the fact that, as shown in FIGS. 7 and 8, eyefatigue increases and 3D motion sickness is likely to occur when theeyes follow very fast in the focal direction. Also, it seems that alimitation should be placed also on the distant position of subtitles,because very distant subtitles cause an unnatural impression due to thesubtitle size with respect to the background as shown in FIG. 6. Alimitation will be necessary also on the near side close to the eyes.This is because positions very near to the eyes involve larger amountsof angular variations of the eyes due to the viewing angle, resulting inan increased degree of eye fatigue. Also, a larger amount of protrusion(in this sense, a representation that is seen as if it protruded veryclose to the viewer, for example) may “surprise” or “startle” theviewer. Also, it is desirable to place stricter limitations when the TVdisplay has a larger screen, because the eyes have to move more busilyin the plane direction, and also because the psychological effects like“surprising” and “startling” also increase. When the reproducingapparatus and the TV are connected in a linked manner, information aboutthe TV screen size is exchanged with the reproducing apparatus, so as toincrease the limitations on the range of protrusion of subtitles etc.When a plurality of streams with different amounts of protrusion areprovided, the system may be configured such that a stream with a smalleramount of protrusion can be selected when the TV screen is larger, and astream with a larger amount of protrusion can be selected when the TVscreen is smaller, for example. Also, when the amount of protrusion canbe varied by a setting on the equipment side, as will be describedlater, the setting may be automatically made by considering TV sizeinformation, the user's conditions (age etc.) and the like.

It is then possible to define stereoscopic video parental levels basedon the above-described evaluation value and the maximum amount ofprotrusion that corresponds to the viewing angle, and then an age limitfor viewing may be set, or a warning against danger may be given toelderly or sick people, according to the stereoscopic video parentallevel. For example, the stereoscopic video parental levels may includeLevel 1 indicating severe fatigue and danger, with the evaluation value(the degree of eye fatigue)>c, the amount of maximum protrusion>d, andthe ordinary parental level being high. Level 2 would indicaterelatively severe fatigue and danger, with the evaluation value (thedegree of eye fatigue)>c, the amount of maximum protrusion>d, and theordinary parental level being normal or lower, or with the evaluationvalue (the degree of eye fatigue)>e, the amount of maximum protrusion>fand the ordinary parental level being high. Level 3 would indicateintermediate fatigue and danger, with the evaluation value (the degreeof eye fatigue)>e, the amount of maximum protrusion>f, and the ordinaryparental level being normal or lower. Level 4 would indicate no fatigueand no danger, with the evaluation value (the degree of eye fatigue)>g,the amount of maximum protrusion>h, and the ordinary parental levelbeing normal or lower.

In the above-described example of stereoscopic video parental levels,there is a relation of c>e>g and a relation of d>f>h, and the ordinaryparental levels (flat-video parental levels) indicate the restrictionson viewing for ensuring safety that are defined about horror movies andthe like in existing 2D video DVDs etc. It will be useful when thesetting of such stereoscopic video parental levels can be determined orchanged at the time of purchase or at the time of initialization, andcan be cancelled and changed at a later time by using an identificationnumber etc.

Second Preferred Embodiment

Next, a second preferred embodiment will be described referring to thedrawings. When stereoscopic images utilizing parallax information, asshown in FIGS. 1 to 3, are intactly broadcasted on television, they aredisplayed like superimposed images. They cannot be viewed without adedicated stereoscopic display apparatus constructed as described above.Thus, the broadcasting of stereoscopic video is limited byinfrastructural facilities of equipment on the viewer side, and it isnecessary to provide a dedicated channel that is not intended to be seenby people in general, or to superimpose a flag on the broadcastedinformation to indicate that it is 3D. Accordingly, it is generallyconvenient to distribute such stereoscopic video in the form of arecording medium, and to reproduce it with a dedicated player or with aplayer having this function. Considering such circumstances, methods andformats for storing stereoscopic video in a recording medium will now bedescribed.

FIG. 10 shows a recording medium 26 according to the second preferredembodiment. The recording media (video media) of this invention can beoptical disk media, such as DVDs, BDs, HD-DVDs, MOs, etc., and can alsobe HDD media, of course. HDDs are usually not portable themselves, butare advantageous in terms of capacity when recording broadcastedstereoscopic video information. On the other hand, optical disk media,such as ROM media, are advantageous for the distribution of stereoscopickiller content before broadcasted or for the distribution of chargeablestereoscopic content. The disk-like recording medium 26 shown in FIG. 10is divided into a region that stores control information about the videoinformation (video control information 23), a region that stores astereoscopic video (a video title 24), and a region that stores anordinary 2D video (a video title 25).

FIG. 11 is a diagram showing an exemplary structure of the video streamin the region of the video title (video content) 24 of FIG. 10. Thevideo title 27 shown in FIG. 11 includes 2D video information 28,user-selectable 2D video information 30 and 31, 3D video information 29that is automatically selected or user-selected when the displayapparatus is capable of displaying stereoscopic images, 2D videoinformation 32 that is reproduced following the video information 29 to31, and 2D video information 33 provided at the end of the video title27. FIG. 11 also shows GOP-layer stream information including asupplementary information region 34 that is located at the head of thefollowing GOP video information 35 and 36 and that describessupplementary information related to the GOP video information, asupplementary information region 37 that is located at the head of GOPvideo information 38 and that describes supplementary informationrelated to the GOP video information, and a supplementary informationregion 39 that is located at the head of GOP video information 40 and 41and that describes supplementary information related to the GOP videoinformation.

FIG. 11 also shows picture-layer stream information including a packetdata portion 42 that describes supplementary information, I picture data43 composed of intra-encoded data, a B picture 44 as encoded datapredicted from the I picture data 43 and P picture 45 in the temporaldirection, and a P picture 45 as encoded data predicted from the Ipicture data 43 only in one temporal direction. FIG. 11 also shows atransport packet data layer including a packet 46 as a transport packetdata portion that describes supplementary information and that isidentical to the packet data portion 42, a first packet 47 carrying theI picture data 43 divided into transport packets, a transport packet 48storing the last data piece of the I picture data 43, a last portion 49of the I picture data in the transport packet 48, and a padded portion50 in the transport packet 48.

FIG. 12 illustrates the hierarchical data structure of the region of thevideo title (video content) 27 where a selection can be made among the3D video information 29 and the 2D video information 30 and 31. FIG. 12shows supplementary information 51 that is located at the head of theregion where the 3D video information 29 and 2D video information 30 and31 are selectable and that stores information related to the videosequence, a GOP video information sequence 52 in this region,supplementary information 53 that stores information related to a GOPvideo information sequence 54, and supplementary information 55 thatstores information related to the OOP video information sequence 56 atthe last of this region. FIG. 12 also shows picture layer streaminformation including I picture data 57 composed of intra-encoded data,a B picture 58 as encoded data that is predicted from the I picture data57 and P picture 59 in the temporal direction, and a P picture 59 asencoded data that is predicted from the I picture data 57 only in onetemporal direction.

FIG. 12 also shows a transport packet data layer including a firstpacket 60 carrying the I picture data 57 divided into transport packets,a transport packet 61 storing the last data piece of the I picture data57, a last portion 62 of the I picture data in the transport packet 61,and a padded portion 63 in the transport packet 61. Also, in FIG. 12,the arrow A indicates a reproduced portion, the arrow B indicates aportion that is skipped when 3D reproduction is performed, and the arrowC indicates a portion that is reproduced for 3D reproduction, whereright-eye GOP video information and left-eye GOP video information areprovided. FIG. 13 illustrates the hierarchical data structure of theregion of the video title 27 where a selection can be made between the3D video information 29 and the 2D video information 31. This structureis basically the same as that of FIG. 12, and therefore like componentsare designated by like reference characters and not described againhere.

Now, as shown in FIG. 10, the structure of data recorded in an opticaldisk or HDD medium includes a region for the video control information23 for recording video-related supplementary information, sequences,etc., and a region for actual video titles (video content) 24, 25. Inthis case, 3D video is not always totally 3D from the beginning to theend, but it may be a mixture of 3D and 2D, or such video streams may beswitchable by user selection. In particular, DVD standards allow fordisplay of user-selectable and switchable video information streams,such as multi-angle, and 3D video information is produced byadditionally constructing a 3D video stream on a 2D video stream, sinceusers' facilities are not always 3D-compatible. When a user's displayequipment is compatible with 3D, it may be automatically recognized,e.g. by the link function of HDMI terminal, and a 3D video stream may bedisplayed selectively, or the user may operate a button to select anddisplay 3D video. Needless to say, some content may be totally 2D or 3D,but consideration should be given to such composite formats.

It is desired that supplementary information regions about the videoinformation stream of the video title 24 be provided also on the videoinformation stream, so as to allow access to the information, managementof the information, switching of settings of equipment, etc. Inparticular, when content includes a mixture of 2D and 3D video streams,it is necessary on the TV side to detect whether the video stream is 2Dor 3D. When supplementary information regions are provided on thestream, the settings on the TV side can be readily and automaticallychanged on the basis of the information. When a player/recorder forreproducing/recording a recording medium controls all settings in aclosed manner, it will be satisfactory to describe the controlinformation only in the video control information 23 where the controlinformation is collectively recorded in a part of the disk. However,when it is connected to a TV, especially when the TV is switched in thecourse of reproduction, superimposing necessary minimum controlinformation on the video information itself allows the settings on theTV side to be automatically switched. When such control information isabsent in the video information, the procedure requires detecting theswitching of TV, separately sending control information from theplayer/recorder, changing the settings on the TV side, and then sendingthe video information. Needless to say, as to the change of settings onthe TV side, it is necessary to provide a mechanism for quickly changingthe settings of the display apparatus, since the stereoscopic videoreproduction processing itself like the switching of polarized light, isconducted on the display apparatus side.

The supplementary information 51 can be used also for access andmanagement of information, and it has become established as Naviinformation in DVD standards. When both 2D video and 3D video exist in amixed manner, they exist in parallel in the time sequence of thecontent, as shown by the 3D video information 29 and 2D videoinformation 30 and 31 in FIG. 11. Accordingly, the first supplementaryinformation 34 should be located at the head of the GOP data informationgroup. Then, by reading the contents of the supplementary informationfirst, it is possible to know whether the information of the next GOPsequence is 2D or 3D, and if it is 3D, it is possible to know whether itis for the left eye or for the right eye, and to know their locations inthe GOP video information group (where to access). The GOP videoinformation group having the supplementary information 51 at the head isdefined as a video unit that is larger than the GOP video information.

In the case of video information data that is compressed also in thetemporal direction, such as by MPEG, the information exists in units ofGOP information having an I picture at the beginning, and thereforeaccess to the video data has to be made in units of GOP videoinformation. Also, the supplementary information, which has to be readin the first place, should be located at the head of the GOP videoinformation group. For example, when a 3D video information portion isreproduced as shown in FIG. 12, the supplementary information 51 isreproduced first (the arrow A in the diagram), and the 2D videoinformation 30 and 31 are skipped, and then the 3D video information 29is reproduced. In this way, the 2D video information 30 and 31 areskipped as shown by the arrow B so that the memory of the reproducingequipment does not capture unnecessary information (the 2D videoinformation 30 and 31 in this case) and an increase of unwanted storageis avoided, and then the 3D video information 29 is reproduced as shownby the arrow C without interruption of video.

At the picture layer underneath, the supplementary information 51 at thebeginning of GOP video information is located before the I picture 57.Also, in the lowermost-layer data, the compressed video data is dividedinto transport packets 60, 61 as shown in FIG. 12, since dividingcompressed video data into transport packets is convenient to providecompatibility with digital broadcasts such as ground-wave, satellite,and cable digital broadcasts. In this case, too, the supplementaryinformation 51 is described in the transport packet at the head of theGOP video information group 52. Needless to say, a private packet thathas been newly defined for transport packets is used. In the transportpacket 61 at the end of the I picture 57, the data does not always endto fit the fixed transport packet unit, and so it is preferable to padthe last portion 63 with “00” or “FF” so that the packet data completesto fit the GOP video information unit. When the sequence branches outinto two streams including one 2D video stream 31 and one 3D videostream 29 as shown in FIG. 13, the arrow B skips a smaller amount of GOPvideo information than in FIG. 12, but the operation is basically thesame as that shown in FIG. 11.

The contents of the supplementary information will be described in moredetail. The supplementary information 51 shown in FIG. 14 includescontent information 64, time code 65, location information 66,information 67 about video information, information 68 about audioinformation, and information 69 about OSD information. The contentinformation 64 shown in FIG. 14 includes content name 70, copyright 71,encryption information 72, presence/absence of 3D images 73, andavailable region information 74.

The time code information region 65 shown in FIG. 15 includespresentation time 65A and synchronization information 65B. The locationinformation 66 shown in FIG. 16 includes seamless information 75, jumpdestination information 76, angle information 77, and in-GOP locationinformation 78. The video information 67 shown in FIG. 17 includesresolution information 79, frame rate information 80, 3D videoinformation 81, parental information 82, angle information 83,encryption information 84, information 85 about 3D video scheme andpresence/absence, 3D video frame rate information 86, number of 3D videoinformation 87, information 88 about depth resolution, information 89about the degree of depth variation, information 90 about permittedsubtitle depth, information 100 about depth limitations, and information101 about parallax limitations.

FIG. 18 is a schematic diagram illustrating display of the supplementaryinformation on a TV as a display apparatus. FIGS. 19A and 19B areschematic diagrams illustrating a multi-angle filming with a pluralityof cameras. The OSD information 69 shown in FIG. 20 includes OSDlocation information 69A, OSD storage location information 69B, font andcharacter size designation 69C, in-plane OSD location information 69D,depth-direction OSD location information 69E, depth position 69F,permitted-depth limitation 69G, and depth zooming speed 69H.

Now, the supplementary information 51 shown in FIG. 14 is firstdescribed for each GOP video information group on the stream, and it issent together with the video information in HDMI transmission to a TVetc. Accordingly, needless to say, it includes information necessary forsettings on the TV side, especially about 3D video display.

Next, the content information 64 shown in FIG. 14 will be described. Thecontent name information 70 may be displayed as OSD information on theTV side, including (1) content name, (2) cast, (3) time of production,(4) distributor, (5) names of relevant works, and (6) summarized story.When the supplementary information 51 superimposed on the video streamis included, it is possible to display the contents of the content nameinformation 70 even when the input on the TV side is switched to 3Dvideo information in midstream.

The copyright information 71 shown in FIG. 14 describes (7) copyrightowner, (8) distributor, (9) importer, and (10) capital participant,whereby the information about the copyright owner of the video streamcan be distributed at the same time, allowing the copyright owner toclaim against illegal use of the reproduced data. Also, since thisinformation is superimposed on the video stream, it is alwaysdistributed to the TV side and the copyright information can bedisplayed even when the TV is changed.

The encryption information 72 shown in FIG. 14 includes (11)presence/absence of encryption and (12) encryption scheme, whereby thedestination equipment can be informed whether the information isencrypted and requires a high level of security, or the informationrequires no security, like commercials.

The 3D video information 73 shown in FIG. 14 describes (13) whetherready for 3D, (14) whether ready for totally 2D display (whether it canbe reproduced to the end only with 2D video display), and (15) when3D-ready, whether 3D video reproduction has priority, wherebyinformation can be displayed to the user of a 3D-incompatible TV to lethim know that his TV is 3D-incompatible. Also, when link-connected witha TV by HDMI, it is possible to automatically switch the TV side to 3D(for example, to automatically display two video streams field by fieldas shown in FIGS. 1 and 3).

When the TV side does not have a 3D video function, measures can betaken by indicating, on the TV or reproducing apparatus side, that theTV is incompatible, or by ejecting the disk, for example.

The available region 74 shown in FIG. 14 can describe (16) a 2D videoreproduction permitted region and (17) a 3D video reproduction permittedregion, whereby it is possible to permit the reproduction of this diskonly in a limited region, and also to permit 2D only and permit 3Dcompatible display only in a limited region. This is because, in somecases, 2D video reproduction only is permitted in a particular regionwhen the conditions for license about 3D video reproduction are notestablished. In a region where 3D video reproduction is not permitted,measures can be taken by allowing even a 3D video display apparatus toreproduce 2D video only, or by ejecting the disk, for example.

Next, the time code information 65 shown in FIG. 15 will be described.With video content that contains a mixture of 2D and 3D, even when theuser gives an instruction for switching in midstream (from 3D to 2D, forexample), it is necessary to continuously reproduce the videoinformation without interruption or omission. Also, the user may give aninstruction for a time search to move forward or backward, e.g. to 10minutes before. It is therefore necessary to record the presentationtime 65A at the beginning of the GOP video information group, so as toprovide reproduction time information from the start of the video title.It is possible to display the remaining time on the TV side, byrecording information about the remaining time to the end of thereproduction of the title, or the total title reproduction time.

Also, 3D video is likely to cause eye fatigue etc. as described in thefirst preferred embodiment. Accordingly, it is possible to give aninstruction to take a break to prevent eye fatigue, or to warn about thedanger, by displaying a time code from the start of 3D videoreproduction (how long 3D video has been viewed continuously), or bydisplaying how long 3D video has been viewed in total in this videocontent. Also, when there are right-eye and left-eye GOP videoinformation for 3D, it is possible to make field designationcorresponding to the order of reproduction. That is, the presentationtime 65A describes (18) a time code from the start of the title(presentation time), (19) information about the remaining time to theend of the reproduction of the title, or a total title reproductiontime, (20) a time code from the start of 3D video reproduction (3Dpresentation time), (23) total 3D reproduction time, and (24) the orderof reproduction of right-eye and left-eye images or field designation.The synchronization information 65B defines the synchronization of thevideo content, by the order of reproduction of right and left images orby field designation.

Next, the location information 66 shown in FIG. 16 will described. Inparticular, when video content includes both 2D and 3D in a mixedmanner, it is necessary to describe location information about GOP videoinformation groups, in order to skip information unnecessary forreproduction or to move to the beginning of required data. Also, in thecase of special reproduction operation, it is necessary to consider therequirement that access has to be first made to an intra-compressedpicture, because of the characteristics of temporally-compressed video,such as by MPEG. Accordingly, the seamless information 75 records (25)absence/presence of seamless reproduction (to the next GOP videoinformation group). Also, the jump destination information 76 records(26) jump destinations' (forward and backward) address 1, address 2,etc., (27) jump destinations' time code information 1, time codeinformation 2, etc. (having multiple pieces of jump destinationinformation as table information), and (28) presence/absence of 3D videoinformation at jump destinations. When the information of (28)presence/absence of 3D video information at jump destinations indicatesthe absence of 3D video information at the jump destination, the settingof the TV can be returned to 2D during the reproduction of the stream.

The angle information 77 records (29) GOP video information address 1,address 2, etc. that correspond to a plurality of angles, and (30) timecode information 1, time code information 2 etc. about the GOP videoinformation corresponding to a plurality of angles. The in-GOP locationinformation 78 records (31) address information 1, address information 2etc. as location information about P pictures in each GOP. Thus, thepresence of the seamless information 75 enables sequential reproductionby connecting necessary angles, and the location information in GOPsenables fast forward and fast reverse reproduction by reproducing Ipictures only or I and P pictures only.

Next, the video information 67 shown in FIG. 17 will be described. Inthe video information 67, the pieces of information shown below areespecially necessary as stereoscopic video information. First, theresolution information 79 records (32) a resolution of 2D videoreproduction (in-plane direction), a resolution of PinP images (in-planedirection), and (33) a resolution of 3D reproduction (in-planedirection). The 3D video presence/absence and scheme 85 records (34)presence/absence of 3D and (35) designation about the 3D video scheme(scan-doubling rate designation, presence/absence of the use ofpolarizing eyeglasses, presence/absence of the use of a liquid-crystalshutter mechanism). The 3D frame rate 86 records (36) a 2D videoreproduction frame rate and a 3D video reproduction frame rate.

The number of 3D video information 87 records (37) the number ofindependent 3D video information streams that are reproduced inparallel. When there are n different angles, it is described as n=anglenumber. On the basis of this information, the number of angles isdisplayed during reproduction, and it is possible to switch the angle byuser selection, and to recognize the angle numbers through display. Thenumber of 3D video information 87 also records (38) the number of videostreams and camera information about right and left images that aresequentially switched. For example, as shown in FIG. 19A, when imagesare taken by using cameras D to H having five shifted parallaxes, orwhen an animation is recorded by CG as five pieces of parallactic videoinformation, this number and the intervals between the cameras or theirangles are described. For example, the supplementary information isdescribed as “general information—the number of video streams isfive—the camera interval is ** mm”, “camera D's image 1—angle 1”,“camera E's image 2—angle 2”, “camera F's image 3—angle 3”, “camera G'simage 4—angle 4”, and “camera H's image 5—angle 5”.

When there are five video streams with shifted parallaxes, it ispossible, as shown in FIG. 19B, to reproduce four stereoscopic anglevideo streams with slightly different angles with the five pieces ofparallactic video information, with the angle D as the left image of thestereoscopic angle 1, the angle E as the right image of the stereoscopicangle 1 and the left image of the stereoscopic angle 2, the angle F asthe right image of the stereoscopic angle 2 and the left image of thestereoscopic angle 3, and the angle G as the right image of thestereoscopic angle 3 and the left image of the stereoscopic angle 4, andthe angle H as the right image of the stereoscopic angle 4. It is alsopossible to rotate images by sequentially shifting the angleinformation. Thus, one video stream does not indicate one angleinformation, but new angle information can be constructed by combiningimages having adjacent parallactic angles. In recent years, with theprogress of CG techniques, it is easy to produce stereoscopic images inanimation, and the point of view can be changed by shifting the angle,by preparing parallactic information of multiple angles and accessingthe information according to the user's indication from a remotecontroller.

The depth resolution 88 records (39) 3D video depth resolution 1,resolution 2, etc. When there are a plurality of 3D video streams, itdescribes a plurality of depth-direction resolutions. For example, in COvideo, when the depth resolution is extremely low and the image does notmove smoothly in time, the depth can be complemented in the temporaldirection on the basis of this information so as to produce smoothdisplay. The degree of depth variation 89 records (40) the degree of 3Dvideo depth variation 1, the degree of variation 2, etc. When there area plurality of 3D video streams, a plurality of depth-directionvariation degrees are described. In particular, the degree of variationis closely related to the fatigue of human eyes as described in thefirst preferred embodiment, and it is recorded to ensure safety, and canbe utilized to give a warning to the user, or to instruct the user totake a break, for example.

The permitted subtitle depth 90 records (41) the range of permittedsubtitle depths (maximum viewing angle 1, minimum viewing angle 1,maximum viewing angle 2, minimum viewing angle 2, etc.). When there area plurality of 3D video streams, a plurality of depth-directionvariation degrees are described. Subtitle information is likely toaffect eye fatigue, because the focus must be frequently adjustedalternately to the subtitles' focus position and the point of attentionduring the viewing of stereoscopic video as described later, and it istherefore necessary to sufficiently limit the range of display. Also,when the depth information is described in terms of real distance, it isdifficult to represent it in numerical form because the far side is atan infinite distance, and therefore it is desirable to describe it interms of viewing angle information. Also, it has no meaning to representnear-infinite distances with detailed numerals, and therefore a lowerlimit may be set, e.g. by omitting viewing angles of 1 degree or less.The player sets the depth positions of subtitles in OSD display on thebasis of these pieces of information.

The depth limitation 100 records (42) depth limitation (maximum viewingangle 1, maximum viewing angle 2, etc.). When a stereoscopic imageprotrudes or moves extremely forward, it will cause a psychologicalimpression like “surprising”. Accordingly, the amount of protrusion ofstereoscopic images, not of subtitles, is limited, so that the imagesare soft for the eyes or so that the images will not surprise the viewertoo much. In this case, in the player, viewing angles corresponding tothe maximum amounts of protrusion are previously recorded in the videocontent as shown in FIG. 17, making it possible, as shown in FIG. 18, togive a warning to young children, or to indicate restrictions on theviewing.

The parallax limitation 101 describes (43) limitations of the amount ofparallax (for filming, maximum viewing angle 1, minimum viewing angle 1,maximum viewing angle 2, minimum viewing angle 2, etc.). When there area plurality of 3D video streams, a plurality of depth-directionvariation degrees are described. This information corresponds to thedistance between two cameras during filming. The amount of referenceparallax differs depending on the interval between human eyes, and thisinformation defines the range of reference angles. It is thus possibleto previously grasp the degree of discomfort that a small child, havinga short interval between the eyes, would feel when viewing the video.

Thus, it is possible to alleviate discomfort by preparing video contentthat contains a plurality of stereoscopic video streams with differentreference parallax values so that a selection can be made on the basisof the interval between the viewer's eyes. In animation and the like,recent innovative CG techniques are capable of easily changing thereference parallax value with a computer. In this case, such parallaxvalue limitation information is described in the supplementaryinformation, whereby the player can provide select buttons, e.g.indicating “** (age)”, “** (age)”, and “adult”, as shown in FIG. 18, andthen a selection can be made with the buttons so that the referenceparallax value of the original video content can be adapted to theviewer to present proper stereoscopic viewing. It is also possible toavoid eye fatigue etc. that would be caused by the viewing of video withimproper parallax for a long time. Also, the parental 82 defines astereoscopic video parental level for 3D viewing, as well as an ordinaryflat-video 2D parental level. The parental 82 records (44A) a flat videoparental level (a parental level like those defined for existing DVDs)and (44B) a stereoscopic video parental level (a stereoscopic videoparental level as described in the first preferred embodiment).

As shown in FIG. 20, the OSD information 69 records, first, locationinformation 69A as supplementary information about the OSD itself, andOSD information storage location 69B describing addresses where theinformation about the OSD itself is stored. With the OSD display 69, thesupplementary information is first captured and understood by amicrocomputer etc., and then the actual OSD is obtained and displayed onthe basis of the storage location information.

The information 69C about character size etc. records (45) characterfont, character size. The in-plane location information 69D records (46)character location information (X position, Y position).

The depth-direction OSD location information 69E records (47) depthposition 69F, (48) permitted-depth limitation 69G (limitations foralleviating eye fatigue as described in the first preferred embodiment,such as a far-side limit position, a near-side limit position, alimitation on the amount of depth variation, etc.), and (49) depthzooming speed 69H (presence/absence of zooming, zooming speed). Thedepth zooming speed 69H defines zooming to alleviate eye fatigue, bygradually zooming from a certain subtitle to the next subtitle, insteadof instantaneously varying the depth position.

The above-described pieces of supplementary information from (1) to (49)about 3D video information are superimposed on the video informationstream and distributed together with the video information, and similardescriptions can be recorded also in the video control information 23 ina region separate from the video information itself as described below.This allows all information to be read at the time when the player isstarted, and thus enables various initializations to be made. This alsoenables descriptions of more detailed control information, since alarger amount of information can be described regardless of bit rate andmemory limit, than when it is superimposed on the video information.

Next, the structure of control information recorded in a certain regionon the recording medium separately from the video information will bedescribed. FIG. 21 is a diagram illustrating the details of a GOP tableportion of collectively recorded video control information 23, and FIG.21 also shows the video-related information recorded therein. The videocontrol information 23 shown in FIG. 21 includes content information 64,copyright 71, encryption information 72, presence/absence of 3D images73, available region information 74, GOP table information 102, sequenceinformation 103, menu information 104, and menu OSD data 105. The GOPtable information 102 is described in the form of a table as shown inFIG. 21, having sections for GOP number, logical address, time code,sequence, location, video, audio and OSD.

In particular, FIG. 21 shows the structure of the video section, whichincludes resolution information 79, frame rate information 80, 3D videoinformation 81, parental information 82, angle information 83, andencryption information 84. FIG. 21 also shows that the 3D videoinformation 81 includes 3D scheme and presence/absence information 85,3D video frame rate information 86, the number of 3D video information87, depth resolution information 88, depth variation degree information89, permitted subtitle depth information 90, depth limitationinformation 100, and parallax limitation information 101.

FIG. 22 is also a diagram illustrating the GOP table portion of thecollectively recorded video control information 23, and FIG. 22 showsthe details of the video-related information recorded therein. Inparticular, FIG. 22 shows the structure of the OSD section, includingpresence/absence of subtitles 106 and OSD information 69. The OSDinformation 69 includes OSD location information 69A and OSD storagelocation information 69B, where the OSD location information 69Aincludes font and character size designation 69C, in-plane OSD locationinformation 69D, and depth-direction OSD location information 69E, andthe depth-direction OSD location information 69E includes depth position69F, permitted-depth limitation 69G, and depth zooming speed 69H.

FIG. 23 illustrates the structure of the sequence information in thevideo control information collectively recorded in a region of therecording medium separately from the video information, and FIG. 23illustrates that the sequence information 103 is recorded in the form ofa table. The video section of FIG. 23 includes resolution information79, frame rate information 80, angle information 83, 3D videoinformation 81, and parental information 82. The 3D video information 81includes 3D video scheme and absence/presence information 85, 3D videoframe rate information 86, the number of 3D video information 87, depthlimitation information 100, and parallax limitation information 101. TheOSD section of FIG. 23 records presence/absence of subtitles 106,subtitle font, color 107, subtitle display scheme 108, subtitle depthlimitation 109, and subtitle data address 110.

The control information located in a certain region of the recordingmedium separately from the video information describes all informationincluding the supplementary information 34, 51 that are superimposed onthe video information stream. By this, the control information is readfirst when the player/recorder is started so that variousinitializations can be made.

First, the video control information 23 is described as shown in FIG.21. Like the supplementary information 51 superimposed in the videoinformation of FIG. 14, it includes content information 64, time code65, location information 66, information 67 about video information,information 68 about audio information, and information 69 about OSDinformation. However, in the video control information 23 where a largeramount of information can be stored, it is possible to describe tableinformation about all GOPs, like the GOP table 102, and so the contentsof the information can be grasped in units of GOP video informationwithout reproducing the video. The GOP table 102 is described like thetable of FIG. 21, including logical addresses, and so data fileidentification information can be detected from a signal read from agiven sector region, by utilizing the logical addresses. Then, on thebasis of the detected data file identification information, the positionof a data file recorded on the disk medium is identified. The data filecorresponds to an encoded unit located at the position indicated by theposition identification signal. Then, the data file is read on the basisof the identified position on the disk medium, and the video signalencoded in the unit in the read data file is decoded, and can thus bereproduced. Thus, for reproduction, the position where a certain videosignal encoded in the unit is recorded can be readily andinstantaneously specified at a desired point of time.

The supplementary information about video in the GOP table 102 includes3D video information, and the same items as the items (32) to (43)described in the video stream can be described for each piece of GOPvideo information. Also, as shown in FIG. 22, the subtitle informationdescribes the presence/absence of subtitles 106, the OSD information 69(OSD location information 69A˜depth zooming speed 69H), and thus thesame information as the information (44) to (49) can be described assupplementary information about subtitles for each piece of GOP videoinformation.

Also, as shown in FIG. 23, the video control information 23 can alsodescribe sequence information for each piece of GOP video information.Thus, at the start of reproduction, an address is generated about acertain sector region where information about the order of data filereproduction is recorded, and the reproduction order information is readfrom the data read by data reading means, so as to detect thereproduction order information, and then the order of sector addressesare generated on the basis of the reproduction order information. It isthus possible to reproduce video signals recorded in sectors that aredistributed on the recording medium. The sequence table information canalso describe the supplementary information shown as (32) to (49).

Third Preferred Embodiment

Next, a third preferred embodiment will be described. FIG. 24 is a blockdiagram of a stereoscopic video recording apparatus according to thethird preferred embodiment. The stereoscopic video recording apparatusshown in FIG. 24 includes AD converters 146 for digitizing video signalsrespectively for the right-eye and left-eye images of stereoscopic videoutilizing parallax information, motion vector detectors (motiondetectors) 147 necessary for video compression in the temporaldirection, DCT transform circuits 148 necessary for intra-compression,adaptive quantization circuits 149 necessary for intra-compression, andinverse quantization circuits 150 for local decoders. The stereoscopicvideo recording apparatus of FIG. 24 also includes variable-lengthcoding circuits 151 necessary for intra-compression, inverse DCTtransform circuits 152 for local decoders, frame memories 153 for localdecoders, a buffer memory 154 for storing data after compressed, an OSDinformation encoder 155, an audio encoder 156, a format encoder 157,modulating means 158 for generating a signal to be written to an opticaldisk 165, and an LD modulation circuit 159. The stereoscopic videorecording apparatus of FIG. 24 further includes an address headeridentification circuit 160 for extracting addresses for recording to theoptical disk 165, a reproduction amp 161 for reproducing a signal froman optical head 164, a servo circuit 162 for controlling the opticalhead 164, feed motor 163 and rotary motor 166, and a system controller167 for controlling and managing the sequence of the entire apparatus.

FIG. 25 is a diagram illustrating the stream structure of a stereoscopicvideo signal generated on the basis of the stereoscopic video recordingapparatus of FIG. 24. The stream structure of the stereoscopic videosignal of FIG. 25 includes supplementary information 51, which isfollowed by repetitions of a right-eye image GOP 168, left-eye image GOP169, audio information 68, and OSD information 69. The right-eye GOP 168of FIG. 25 includes a GOP header 170, picture headers 171 and 173, Ipicture data 172, and B picture data 174. Also, the GOP header 170 ofFIG. 25 includes a user data start code 175 in the GOP header 170, and3D video information 176 in the GOP header 170. Also, the 3D videoinformation 176 of FIG. 25 includes information 177 indicating whetherthe pictures are for the left eye or the right eye, 3D video scheme andpresence/absence information 85, 3D frame rate information 86, thenumber of 3D video information 87, depth limitation information 100, andparallax limitation information 101.

FIG. 26 illustrates a lower-order structure of the stereoscopic videosignal stream generated on the basis of the stereoscopic video recordingapparatus of FIG. 24. FIG. 26 shows the right-eye GOP 168 with transportstream packets 179 to 182, and the left-eye GOP 169 with transportstream packets 185 to 188. The data pieces 183 and 189 are the last datapieces respectively of the transport packets 182 and 188 where the lastdata pieces of the GOP video information are described, and paddedportions 184 and 190 are attached respectively to the data 183 and 189.

Now, the stereoscopic video recording apparatus shown in FIG. 24performs identical video compression for each of the left and righteyes, where the AD converter 146 digitizes right-eye pictures and themotion detector 147 extracts motion vectors on amacroblock-by-macroblock basis. Since the beginning of video data isintra-encoded, it is DCT-transformed in the DCT transform circuit 148,quantized in the adaptive quantization circuit 149,variable-length-coded in the variable-length coding circuit 151, andsent to the buffer memory. In this process, the video data afteradaptive-quantized is processed in the local decoder formed of theinverse quantization circuit 150 and the inverse DCT transform circuit152, and the original video signal is thus reconstructed and stored inthe frame memory 153. Then, by making a comparison withmotion-compensated pictures in the frame memory 153, the followingpictures to be compressed in the temporal direction can be compressed byusing differential information only. Such a compression scheme is awidely-used, basic scheme in compression methods like MPEG, H.264, etc.

In the configuration of FIG. 24, the right-eye images and left-eyeimages are inputted as independent video streams and encoded in separateencoder blocks. Accordingly, the configuration of FIG. 24 includesidentical blocks arranged in parallel for the right eye and the lefteye. However, the same processing can be achieved with a single encoderblock, by providing a memory at the input to once store left-eye andright-eye images, and processing them in the same encoder block at adoubled rate. The stereoscopic video information thus encoded is sent tothe buffer memory 154, and provided with OSD information from the OSDencoder 155, audio information from the audio encoder 156, andsupplementary information necessary for the format from the formatencoder 157, and thus formed into a data format for recording to theoptical disk 165 as a recording medium. The format encoder 157 alsoprovides the supplementary information necessary for the recording of 3Dvideo information according to the present invention, as well as Naviinformation and menu information necessary for conventional optical diskformats.

The video data in the format for recording to the optical disk is sentto the modulating means 158, and optically modulated and provided witherror correcting code as information for physical writing to the opticaldisk 165. Then the LD modulation circuit 159 generates a signal formodulating the laser provided in the optical head 164. In this process,the servo circuit 162, for ensuring stable recording to the optical disk165, controls the feed motor 163 for moving the optical head 164, therotary motor 166 for rotating the disk 165, and the objective lensactuator in the optical head 164, to achieve tracking and focusing.During recording, it is necessary to read addresses on the optical disk165, and a signal received at the optical head is photo-electricallyconverted to a fine signal, and the reproduction amp 161 reproduces thefine signal, and the address header identification circuit 160 generatesaddress information. The address information is sequence-processed inthe system controller 167 together with settings for activation ofindividual blocks, and processings for writing timing etc., especiallyrequiring high-speed timing, are performed in dedicated hardware, andsequence setting portions requiring programming are performed in the CPUetc.

Now, the video stream generated by the stereoscopic video recordingapparatus has a structure as illustrated in FIG. 25. First, generallycompressed video data that includes temporally compressed pictures, e.g.by MPEG, includes intra-compressed encoded pictures called GOP. Forexample, it is formed as a video block of about 15 pictures. Herein,there are two video blocks for the right eye and the left eye thatutilize parallax information, and so the right-eye GOPs 168 and theleft-eye GOPs 169 are sequentially arranged following the supplementaryinformation 51 located at the beginning. In this example, one GOP isused for each of the right eye and the left eye, but a plurality of GOPsof the same number may be used in a range where video conditions areunchanged. Also, the supplementary information 51 is structured asdescribed in the second preferred embodiment, and a user data start codeis newly defined in the GOP header portion, and 3D video information 176may be described following it.

The 3D video information 176 first includes information (or a flag) 177indicating whether it is for the left eye or the right eye, and furtherincludes 3D video scheme presence/absence 85, 3D video frame rate 86,number of 3D video information 87, depth information 100, and parallaxlimitation information 101, as described in the second preferredembodiment. Also, as shown in FIG. 26, for the supplementary information51, a private packet (TSP1) 178 is provided at the transport packetlayer, and it is separated and extracted in the transport decoderportion. Structuring the information as shown in FIG. 25 enablesextraction of the same 3D information also at the MPEG data hierarchylevel. When the reproducing apparatus and a TV are link-connected and 3Dvideo settings are automatically made, or when the TV is switched in thecourse of reproduction, the supplementary information 51 is valid whenthe supplementary information extraction setting is valid on the TVside, and the 3D video information 176 is valid when stereoscopic videosupplementary information at the MPEG layer is valid.

FIG. 25 describes the video stream in units at the MPEG informationlayer, and FIG. 26 describes it in units of transport packets that areone layer below. In FIG. 26, the video data in a block unit of theright-eye and left-eye GOPs 168 and 169 starts at the head of thetransport packet 179, and it does not always fit an integer multiple ofa transport packet at the end of the GOP video information. Accordingly,the remaining part of the last transport packet 182 is padded, and thepadded data 184 is attached to the last data 183 to fill in information.Thus, the right-eye GOP 168 alone can be extracted by extracting thepart from the transport packet 179 to the transport packet 182. Theleft-eye GOP 169 is processed in the same way as the right-eye GOP, andthe left-eye video alone can be extracted by extracting the part fromthe transport packet 185 to the transport packet 188 including thepadded data 190.

The stereoscopic video recording apparatus shown in FIG. 24 encodes theintact left-eye and right-eye images. However, basically, right-eyeimages and left-eye images are visual information shifted by parallax,and therefore they are very closely correlated. Therefore, the amount ofentire information can be compressed by using a configuration thatrecords only the differential information about the left-eye images inrelation to the right-eye images, for example. FIG. 27 is a blockdiagram of a stereoscopic video recording apparatus configured for thispurpose. In the stereoscopic video recording apparatus of FIG. 27, theright-eye video block is structured for primary images, and it adoptsthe same configuration as the video compression blocks shown in FIG. 24.However, for the left-eye images, a differential is taken between theoutput of the motion detector 147 for the left eye and the output of theframe memory 153 for the right eye, so as to solely extract thedifferential information between the right-eye and left-eye images. Thedifferential information is processed in the DCT transform 148 and theadaptive quantizer 149 in the left-eye image line, and further processedin the variable-length coder 151, and then the differential informationfor the left eye is recorded in the buffer memory 154. The followingprocedures, to the writing to the optical disk, are performed in thesame way as shown in FIG. 24. The example of FIG. 27 uses right-eyeimages as primary images and left-eye images as secondary images asdifferential, but the right-eye images and the left-eye images can ofcourse be used in the opposite manner (left-eye images can be used asprimary images and right-eye images can be used as secondary images asdifferential).

The stereoscopic video recording apparatus of FIG. 27 takes adifferential between the left-eye and right-eye images to furthercompress the amount of video information on one side, but theinformation can be further compressed since the video information isbased on parallax information. FIGS. 28A and 28B are schematic diagramsillustrating the principles of parallactic images. FIG. 28A shows anobject 126G on the front side at a depth position 126M, an object 126Hon the depth side at a depth position 126L, a depth position (mostforward) 126N to the eyes 126F, a viewing angle 126D for the object126H, and a viewing angle 126E for the object 126G. FIG. 28B shows aleft-eye image 126GA of the object 126G, a right-eye image 126GB of theobject 126G, a parallax 126I between the left-eye image 126GA and theright-eye image 126GB, and same pixel points 126K in the left-eye andright-eye images. Also, FIG. 28B shows a left-eye image 126HA of theobject 126H, a right-eye image 126HB of the object 126H, a parallax 126Jbetween the left-eye image 126HA and the right-eye image 126HB, and samepixel points 126L in the left-eye and right-eye images.

FIG. 29 is a block diagram of a stereoscopic video recording apparatusfor further compressing images on one side. The stereoscopic videorecording apparatus shown in FIG. 29 includes a parallax informationcalculating circuit 191A for calculating the amount of parallax betweenright-eye and left-eye images, a depth-direction motion detectingcircuit 191C, and an estimated parallax information generating circuit191D. Also, the stereoscopic video recording apparatus of FIG. 29includes an inverse parallax calculating circuit 191B forconverting-original left-eye images to right-eye images on the basis ofthe estimated parallax information, a DCT transform circuit 191E forperforming DCT transform on the basis of the result of a comparison of aright-eye image with the right-eye image generated by the inverseparallax calculating circuit 191B, an adaptive quantization circuit191F, and a variable-length coder 191G. The portion from the parallaxinformation calculating circuit 191A to the variable-length coder 191Gforms a block that performs stereoscopic video compression processing.

FIG. 30 is a diagram illustrating the encoding and decoding of a videostream by the compression method of the stereoscopic video recordingapparatus of FIG. 29. FIG. 30 shows right-eye images 192A to 192G,parallax information calculated value 193 about the left-eye andright-eye images, left-eye images 194A to 194G, depth-direction motionvector calculated value 196, estimated parallax information 197, andcompressed right-eye primary images 195A to 195G. FIG. 30 also showsreproduced primary images 198A to 198G based on the compressed right-eyeprimary images 195A to 195G, estimated parallax information 204 to 210respectively corresponding to the reproduced primary images 198A to198G, and reproduced secondary images 211 to 217. FIG. 31 is a diagramillustrating the structure of a video stream generated by thecompression method of the stereoscopic video recording apparatus shownin FIG. 27 or FIG. 29. The video stream structure shown in FIG. 31 isbasically the same as that of FIG. 25, but it includes differentialvideo information 218 in place of the left-eye GOPs 169, andadditionally includes information 219 in the 3D video schemepresence/absence 85 to indicate whether the information is GOP videoinformation or differential information. FIG. 32 illustrates the streamstructure at the transport level of the video stream generated by thecompression method of the stereoscopic video recording apparatus of FIG.29. FIG. 32 is basically the same as FIG. 26, but it includesdifferential video information 218 in place of the left-eye GOPs 169.

Now, FIG. 28A illustrates the stereoscopic direction with right-eye andleft-eye images utilizing parallax. In FIG. 28A, the parallactic angleseen from the eyes 126F appears different depending on the depth.Accordingly, in the right and left parallactic images shown in FIG. 28B,the object 126G on the near side is seen as being larger, and theleft-eye image 126GA and the right-eye image 126GB are seen as beingseparated away, and the amount of parallax 126I is larger. On the otherhand, the object 126H at a distance is seen as being smaller, and theleft-eye image 126HA and the right-eye image 126HB are seen as beingcloser, and the amount of parallax 126J is smaller.

Accordingly, with information about the amount of parallax (126I or126J) or with information about the parallactic angle (126D or 126E), itis possible, as shown in FIG. 28B, to estimate the right-eye image fromthe left-eye image (126K and 126L: image generation by conversion withparallax information). This estimation assumes that the brightness andcolor do not change depending on the angle of view, and thereforevariations caused by turning-around of images, shades, etc. remainunpredictable by this estimation.

Now, in the stereoscopic video recording apparatus shown in FIG. 29, theparallactic angle is extracted on the basis of in-plane positioninformation about objects that is obtained from the left-eye andright-eye motion detectors 147, and the parallax information calculatingcircuit 191A calculates parallax information on amacroblock-by-macroblock basis or a pixel-by-pixel basis. Also, for thecompression in the temporal direction, the depth-direction motiondetecting circuit 191C extracts motion vectors in the depth directionbetween pictures. The estimated parallax information generating circuit191D generates estimated parallax information with the depth-directionmotion information and the parallax information. Also, as mentionedabove, when there is only the parallax information from an image on oneside (described herein as a right-eye image), it is not possible toperfectly recover the image on the opposite side (described herein as aleft-eye image), and therefore there remains unpredictable information,like variations caused by turning-around of images (e.g., a hiddenportion becomes viewable).

Accordingly, in the stereoscopic video compression by the stereoscopicvideo recording apparatus shown in FIG. 29, the inverse parallaxcalculating circuit 191B first performs a local-decode reproduction ofan image on the opposite side (described herein as the left-eye image),though not perfectly, by utilizing the estimated parallax information,and then a differential is obtained between it and the actually filmedimage that has been compressed (the image in the frame memory 153 of thelocal decoder). The differential information is about theun-reproducible portion caused by variations due to turning-around ofimages as mentioned above, and it is possible to cover the portion thatcannot be perfectly reproduced with the parallax information, in thecompressed stream that utilizes parallax information. Also, though notshown, when depth-direction motion vectors are extracted, the amount ofvariation of parallax is also utilized as information. Thus, thedepth-direction motion vectors can be utilized to increase thecompression efficiency, by using an inverse quantization circuit, aninverse DCT transform circuit, and a frame memory, like an ordinarylocal decoder for information compression.

Now, FIG. 30 shows the image data in units of pictures. In FIG. 30, theright-eye primary images 192A to 192G are extracted from the right-eyecamera, and the left-eye secondary images 194A to 194G are extractedfrom the left-eye camera. Then, the parallax information calculatingcircuit 191A of FIG. 29 calculates the parallax information 193 from theright-eye primary images 192A to 192G and the left-eye secondary images194A to 194G. The depth-direction motion vector calculation 191Cextracts the depth-direction motion-vector calculated value 196 fromvariations of the parallax information 193 between pictures, and theestimated parallax information 197 is generated. The estimated parallaxinformation 197 itself may be on a macroblock-by-macroblock basis or apixel-by-pixel basis.

On the other hand, in the reproduction of the images, the right-eyeprimary images 192A to 192G are in the form of the compressed right-eyeprimary images 195A to 195G that are encoded by the video compression.Specifically, the compressed right-eye primary images include anintra-compressed I picture 195A, P pictures 195D and 195G compressed inthe temporal direction with in-plane motion vectors, and B pictures195B, 195C, 195E, and 195F. The compressed right-eye primary images arereproduced into the reproduced right-eye primary images 198A to 198G bya common compressed-video-decompressing circuit. Then, the reproducedsecondary images (left-eye images) 211 to 217 are restored on the basisof: the reproduced primary images (right-eye images) 198A to 198G; theestimated parallax information 204 to 210 about individual pictures; anddifferential information about individual pictures that areinverse-quantized and inverse-DCT-transformed. The image portionsrestored from the differential information work to complement theportions that cannot be reproduced with the estimated parallaxinformation, caused by turning-around of images and the like.

FIG. 31 illustrates the video stream using the video compression by thestereoscopic video recording apparatus of FIG. 29 or FIG. 27, and it isnecessarily provided as data in GOP video information units. This isbecause the right-eye images are originally in units of GOP videoinformation, and it is necessary to match the unit level of the imagessince the differentially-compressed data for the left eye utilizes theright-eye images. Now, the GOP header 170 in the right-eye GOP 168includes supplementary information related to the stereoscopic video,like those described with FIG. 25. However, as shown in FIG. 31, for theleft-eye images, it is necessary to describe the information 219 in the3D video scheme presence/absence 85 to indicate whether it is GOP videoinformation or differential information, so as to indicate whether it iscompressed information that utilizes the estimated parallax information197 shown in FIG. 29, or differentially-compressed information as shownin FIG. 27, or information that is not compressed as stereoscopic imagesas shown in FIG. 24. FIG. 32 illustrates the stream structure at thetransport packet level, where the end of the GOP video data is padded ina transport packet in a manner as shown in FIG. 26, and the end of thestereoscopically compressed video data, too, is padded in a transportpacket. The description above assumes the right-eye images as primaryimages and the left-eye images as secondary images that are compressedin the stereoscopic direction, but there is no problem at all when theimages are processed in the opposite relation. Also, the video streammay include a mixture of right-eye images as primary images and left-eyeimages as secondary images, and left-eye images as primary images andright-eye images as secondary images. When such mixture is permitted bystandards, it is necessary to describe identification informationindicating which constitutes primary images and which constitutessecondary images.

It is also possible to more simply construct a video stream withstereoscopic images for the left eye and the right eye. For example, thestereoscopic video recording apparatus shown in FIG. 33 includes asynthesizing circuit 220 as a video constructing block for simplyconstructing a video stream. FIG. 34 illustrates the structure of avideo stream by the stereoscopic video recording apparatus shown in FIG.33, where a left-eye and right-eye GOP 221 forms a single GOP videoinformation unit. FIG. 35 illustrates the structure at the transportpacket level of the video stream by the stereoscopic video recordingapparatus shown in FIG. 33.

Now, as shown in FIG. 33, the inputted parallactic images for the lefteye and the right eye are once inputted to the synthesizing circuit 220,and inserted into one picture as two vertically-elongate images. In thisprocess, the pixels of the images are not simply thinned in image lines,but are filtered and compressed in the horizontal direction, and thensynthesized into a left-eye and right-eye picture. Thus, each picture iscomposed of two vertically-elongate images for the left eye and theright eye, and then formed into a stream by a common video compressionscheme. In this case, too, in the video stream shown in FIG. 34, it isnecessary that the supplementary information 51 or the 3D informationregion 176 in the GOP header 170 describe that the video information iscompressed in the horizontal direction, so that it is not intactlyreproduced in an ordinary TV. In the case of the video stream structureshown in FIG. 34, as in those shown in FIGS. 26 and 32, the padded data184 for filling in information is attached to the last data 183 of thetransport packet at the end of the GOP video information at thetransport packet level. The invention has described stereoscopic videorecording apparatuses and stereoscopic video recording methods forrecording on an optical disk, but it is needless to say that entirelythe same effects are obtained when a hard disk is used as a recordingmedium.

Fourth Preferred Embodiment

Next, a fourth preferred embodiment will be described referring thedrawings. This preferred embodiment describes stereoscopic videoreproducing apparatuses, while the third preferred embodiment hasdescribed stereoscopic video recording apparatuses. FIG. 36 is a blockdiagram of a stereoscopic video reproducing apparatus according to thispreferred embodiment. The stereoscopic video reproducing apparatus ofFIG. 36 includes a demodulation and correction circuit 222, an addressheader identification circuit 223, an IF (interface) 224 for connectingthe optical disk drive block and the back-end as a video audioprocessing block, and a data buffer 225 for once storing data from theoptical disk drive block. The stereoscopic video reproducing apparatusof FIG. 36 also includes a system decoder 226 for separating streams ofvideo data, audio data, etc., an MPEG H264 decoder 227 for decompressingcompressed images, an audio decoder 228, an OSD decoder 229 for displayof subtitles etc., and an OSD information depth generating circuit 229A.The stereoscopic video reproducing apparatus of FIG. 36 further includesa 3D video processing circuit 230, a blending circuit 229B forsuperimposing the OSD information on images, a general-purpose IF 231for external connection, a dedicated right-eye IF 232, a dedicatedleft-eye IF 233, a buffer circuit 234, and a system controller 235 forthe entire back-end.

FIG. 37 is a block diagram illustrating a block for decoding left-eyeimages from stereoscopically compressed images, as described in thethird preferred embodiment. The apparatus shown in FIG. 37 includes asystem decoder 236 for extracting parallax information and depth motionvector information from the video stream, an MPEG H264 decoder 237 fordecoding a video stream compressed by MPEG, H 264 or the like, parallaxinformation 238, motion vector information 239, a parallax informationcalculating circuit 240, and a left-eye image reproducing circuit 241.The parallax information 238, motion vector information 239, parallaxinformation calculating circuit 240, and left-eye image reproducingcircuit 241 form a 3D video processing circuit 230.

Now, in the stereoscopic video reproducing apparatus shown in FIG. 36,first, the demodulation and correction circuit 222 in the optical diskdrive reproduces video audio data and supplementary data that aredescribed on an optical disk 165. In this process, a servo circuit 162operates such that the reproduction signal from the optical head 164 iscontinuously extracted with high quality, and the address headeridentification circuit 223 operates such that access can beinstantaneously made to certain addresses. The data reproduced by theoptical disk drive is once inputted to the data buffer circuit 225through the IF circuit 224, and then inputted to the system decoder 226.The system decoder 226 separates the stream into video, audio, etc., andthe audio information is inputted to the audio decoder 228, the OSDinformation is inputted to the OSD decoder 229, and the videoinformation is inputted to the MPEG H264 decoder 227.

In the OSD depth generating circuit 229A, OSD information is generatedas OSD information having depth, on the basis of supplementaryinformation obtained from the system decoder 226. The video streamdecoded in the MPEG H264 decoder 227 is processed as 3D videoinformation in the 3D video processing circuit 230, and blended in theblending circuit 229B with the OSD images having depth, and it can beoutputted from the general-purpose IF, such as HDMI, when the transferrate is low, or the left-eye images can be outputted from the dedicatedleft-eye IF 233 and the right-eye images from the dedicated right-eye IF232.

When images on one side are further compressed by using parallaxinformation, as described in the third preferred embodiment, the 3Dvideo processing 230 of the stereoscopic video reproducing apparatus isconfigured as shown in FIG. 37. In FIG. 37, on the basis of the depthmotion vector 239 and the parallax information 238 as compressedleft-eye image information extracted by the system decoder 236, theparallax information calculating circuit 240 performs a parallaxinformation calculation on a pixel-by-pixel basis or amacroblock-by-macroblock basis, so as to generate conversioncoefficients for generating left-eye images from right-eye images. Byusing the conversion coefficients, the left-eye image reproducingcircuit 241 reproduces left-eye images on the basis of the right-eyeimages generated by the MPEG H264 decoder. When the compressed left-eyeimages are compressed only with parallax information, the processingonly involves re-conversion based on the output of the parallaxinformation calculating circuit 240. On the other hand, in the case ofcompressed information subjected to DCT transform and adaptivequantization in a manner as described with the stereoscopic videorecording apparatus of FIG. 33, it is necessary to provide inversequantization and inverse transform circuits in the left-eye imagereproducing circuit 241.

Next, a stereoscopic reproducing apparatus will be described whichreproduces 2D images, not stereoscopic, from left-eye and right-eyestereoscopic images. FIG. 38A is a block diagram of the stereoscopicreproducing apparatus for reproducing 2D images. The apparatus of FIG.38A includes a synthesizing circuit 242 based on parallax information.FIG. 38B is a diagram schematically illustrating an image constructed bythe synthesizing circuit 242, where a left-eye image 232A and aright-eye image 233A are synthesized into a 2D image 231A. Now, commondisplay apparatuses, such as TVs, are not always 3D-compatible, or theyare more often 2D-compatible. Accordingly, it is desired that a mediumthat records only 3D images can be reproduced also two-dimensionally.The simplest method for this purpose is to reproduce 2D images bydisplaying right-eye images or left-eye images only. For example, when aTV is ready only for 2D, it is automatically detected in the linkconnection processing between the player and the TV, and the images onone side are continuously reproduced.

However, this method is problematic because an image having a depthclose to the eyes (an image that appears protruding) involves a largeamount of parallax, and then the position is shifted to the left orright as shown by the left-eye image 232A or the right-eye image 233Ashown in FIG. 38B. Accordingly, it is possible to reproduce a natural 2Dimage by synthesizing left-eye and right-eye images by using parallaxinformation and reproducing an image in the middle position as shown bythe 2D image 231A. However, the areas near both ends of the screencannot be calculated when the parallax is large, and the areas 241A ofthe image (both ends of the screen) will be “cut”, unless the originalimage is filmed such that the left-eye image is wide to the left and theright-eye image is wide to the right.

Furthermore, a large amount of protrusion (a high degree ofthree-dimensionality) might cause increased eye fatigue or an increased“surprising” impression as described in the second preferred embodiment.To prevent this, FIG. 39A shows a block diagram of a stereoscopic videoreproducing apparatus that is capable of varying the amount ofprotrusion. The apparatus of FIG. 39A includes a left-eye imagereproducing circuit 243 using coefficient-varied parallax, a right-eyeimage converting circuit 244 using coefficient-varied parallax, a userinterface 245 for the varying of the amount of protrusion, and aparallax information coefficient varying portion 246. FIG. 39B is adiagram illustrating the variation of the amount of protrusion by thestereoscopic video reproducing apparatus. Also, FIG. 39C is a diagramillustrating the effect obtained by varying the amount of protrusion bythe circuits of FIG. 39A. FIG. 39D illustrates an OSD bar 246A forvarying the amount of protrusion, which is displayed on a displayapparatus connected to the stereoscopic video reproducing apparatus.

In a scheme in which images on one side are compressed with parallaxinformation, as described with the stereoscopic video recordingapparatus of FIG. 29, parallax information that is directly related tothe amount of protrusion is linked on a pixel-by-pixel basis or amacroblock-by-macroblock basis. Accordingly, when an instruction forvarying the amount of protrusion is given from the user, the instructionis inputted to the user interface 245 by using the OSD display of theOSD bar 246A on the TV screen as shown in FIG. 39D, for example. Then,the parallax information coefficient varying portion 246 determines aconversion coefficient to determine to what degree the amount ofprotrusion should be attenuated. This conversion coefficient determinesthe amount of the parallax calculation in the parallax informationcalculating circuit 240. Then, the left-eye images are reproduced by theleft-eye image reproducing circuit 243 on the basis of thecoefficient-varied parallax, and the right-eye images are converted bythe image converting circuit 244 on the basis of the coefficient-variedparallax information, whereby the amount of parallax between theleft-eye images and right-eye images is converted to be smaller in sucha manner that, as shown in FIG. 39B, the left-eye image 126GA and theright-eye image 126GB are converted from the broken lines to the solidlines. As a result, the stereoscopic images obtained from the outputs ofthe dedicated IFs 232 and 233 are reproduced with a reduced amount ofprotrusion as shown by the stereoscopic triangular object of FIG. 39C.

The stereoscopic video reproducing apparatus of FIG. 39A converts theamount of protrusion by using parallax information, when the parallaxinformation is recorded on the video stream. However, parallaxinformation may be absent. Accordingly, the stereoscopic videoreproducing apparatus shown in FIG. 40 is configured so that the amountof protrusion can be controlled even when parallax information is notrecorded in the video stream. The stereoscopic video reproducingapparatus of FIG. 40 includes MPEG H264 decoders 237A and 237Brespectively for left-eye images and right-eye images, a parallaxinformation extracting portion 247, a right-eye image parallaxconverting portion 248, and a left-eye image parallax converting portion249. In the stereoscopic video reproducing apparatus shown in FIG. 40,the parallax information extracting portion 247 newly detects parallaxinformation from the decoded left-eye and right-eye images. In thiscase, as in the apparatus of FIG. 39A, new parallax information isgenerated in the parallax information calculating portion 240, throughthe processing in the user interface 245 and the parallax informationcoefficient varying portion 246, and it is supplied to the right-eyeimage parallax converting portion 248 and the left-eye image parallaxconverting portion 249.

This preferred embodiment has described apparatuses and methods forreproducing stereoscopic video information recorded on an optical disk,but it is needless to say that entirely the same effects are obtainedwhen a hard disk is used as a recording medium.

While the invention has been described in detail, the foregoingdescription is in all aspects illustrative and not restrictive. It isunderstood that numerous other modifications and variations can bedevised without departing from the scope of the invention.

What is claimed is:
 1. A stereoscopic video reproducing method usingleft-eye images and right-eye images comprising: decoding a primarydigital image block for a primary image; decoding a secondary digitalimage block for a secondary image; and reproducing a stereoscopic imagewith said decoded primary and secondary images, wherein said primaryimage is used as one of a left-eye image and a right-eye image accordingto information indicating said primary and secondary images are used forreproducing said stereoscopic image, and a data flag indicating whethersaid primary image is used as one of a left-eye image and a right-eyeimage, wherein said secondary image is used as the one of the left-eyeimage and the right-eye image for which said primary image is not used,and wherein said secondary image is dependent on image information insaid primary image in such manner that said secondary digital imageblock is derived from said primary digital image block.
 2. Astereoscopic video reproducing apparatus using left-eye images andright-eye images comprising: a decoder which decodes a primary digitalimage block for a primary image, and which decodes a secondary digitalimage block for a secondary image; and a reproducer which reproduces astereoscopic image with said decoded primary and secondary images,wherein said primary image is used as one of a left-eye image and aright-eye image according to information indicating said primary andsecondary images are used for reproducing said stereoscopic image, and adata flag indicating whether said primary image is used as one of aleft-eye image and a right-eye image, wherein said secondary image isused as the one of the left-eye image and the right-eye image for whichsaid primary image is not used, and wherein said secondary image isdependent on image information in said primary image in such manner thatsaid secondary digital image block is derived from said primary digitalimage block.
 3. An optical disc on which is recorded image data used forreproducing a stereoscopic video comprising: a primary digital imageblock for a primary image used as one of a left-eye image and aright-eye image when a stereoscopic video is reproduced; a secondarydigital image block for a secondary image used as the one of saidleft-eye image and said right-eye image for which said primary image isnot used when said stereoscopic video is reproduced; and informationindicating said primary and secondary images are used for reproducingsaid stereoscopic image; and a data flag which indicates whether saidprimary image is a left-eye image or a right-eye image, wherein saidsecondary image is dependent on image information in said primary imagein such manner that said secondary digital image block is derived fromsaid primary digital image block.